System and method for facilitating prediction model training

ABSTRACT

In certain embodiments, training data may be generated for training a prediction model. Training data including first datasets may be obtained, where the first datasets include a plurality of feature types. A determination, via a relevancy model, based on the training data, of whether a feature type satisfies a first condition may be made. If the first condition is satisfied, one or more second datasets may be obtained to update the training data, where the second datasets include the plurality of feature types. A determination, via the relevancy model, based on the updated training data, may be made as to whether the feature type satisfies a second condition. The first and second conditions may relate to whether the feature type has a threshold amount of influence on the prediction model. If the second condition is satisfied, the updated training data may be provided to the prediction model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.No. 16/655,581, filed Oct. 17, 2019, the content of which is herebyincorporated by reference in its entirety.

FIELD

Certain embodiments disclosed herein relate to facilitating predictionmodel training, including, for example, generating training data totrain a prediction model without a feature type of the training datahaving an amount of influence on the prediction model.

BACKGROUND

Training data may be used to train a prediction model. The training datamay include feature types that influence or impact an output of a givenprediction model in different ways. An amount of influence a featuretype has on the prediction model's output may vary. For instance, thetraining data may unknowingly causes the prediction model's results tofavor or not favor a particular feature type or types. However, theprediction model needs to be trained in order to determine whether thefeature types included by the training data influences or impacts theprediction model's results. Therefore, it would be beneficial todetermine whether the training data will influence or impact the resultsof the prediction model to favor a particular feature type or typesprior to the prediction model being trained.

SUMMARY

In some embodiments, training data including first datasets may beobtained. A determination of whether a feature type satisfies acondition may be effectuated via a relevancy model or other model basedon the training data. As an example, the condition may relate to thefeature type not having a threshold amount of influence on a machinelearning model. In response to determining that the feature type failsto satisfy the condition, one or more second datasets may be obtained toupdate the training data such that the updated training data includesthe second datasets. A determination may of whether the feature typesatisfies the condition may be effectuated via the relevancy model orother model based on the updated training data. In response todetermining that the feature type satisfies the condition, the updatedtraining data may be provided to the machine learning model to train themachine learning model.

Various other aspects, features, and advantages of the invention will beapparent through the detailed description of the invention and thedrawings attached hereto. It is also to be understood that both theforegoing general description and the following detailed description areexamples and not restrictive of the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 show a system for facilitating prediction model training, inaccordance with one or more embodiments.

FIG. 2 shows a prediction model trained using training data, inaccordance with one or more embodiments.

FIG. 3 shows a data corpus database, in accordance with one or moreembodiments.

FIG. 4A shows a relevancy model configured to determine a relevance of afeature type based on input training data, in accordance with one ormore embodiments.

FIG. 4B shows the relevancy model configured to determine an updatedrelevance of the feature type based on input updated training data, inaccordance with one or more embodiments.

FIG. 5 shows a flowchart of a method of facilitating prediction modeltraining, in accordance with one or more embodiments.

FIG. 6 shows a flowchart of another method for facilitating predictionmodel training, in accordance with one or more embodiments.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific examples are set forth in order to provide a thoroughunderstanding of example embodiments. It will be appreciated, however,by those having skill in the art that embodiments may be practicedwithout these specific details or with an equivalent arrangement.

FIG. 1 shows a system 100 for facilitating prediction model training, inaccordance with one or more embodiments. As shown in FIG. 1, system 100may include computer system 102, client device 104 (or client devices104 a-104 n), or other components. Computer system 102 may includetraining data subsystem 112, relevancy subsystem 114, model subsystem116, and/or other components. Each client device 104 may include anytype of mobile terminal, fixed terminal, or other device. By way ofexample, client device 104 may include a desktop computer, a notebookcomputer, a tablet computer, a smartphone, a wearable device, or otherclient device. Users may, for instance, utilize one or more clientdevices 104 to interact with one another, one or more servers, or othercomponents of system 100. It should be noted that, while one or moreoperations are described herein as being performed by particularcomponents of computer system 102, those operations may, in someembodiments, be performed by other components of computer system 102 orother components of system 100. As an example, while one or moreoperations are described herein as being performed by components ofcomputer system 102, those operations may, in some embodiments, beperformed by components of client device 104. It should be noted that,although some embodiments are described herein with respect to machinelearning models, other prediction models (e.g., statistical models orother analytics models) may be used in lieu of or in addition to machinelearning models in other embodiments (e.g., a statistical modelreplacing a machine learning model and a non-statistical model replacinga non-machine-learning model in one or more embodiments).

In some embodiments, system 100 may generate training data for aprediction model. Prior to using the training data to train theprediction model, the training data may be analyzed via relevancy modelto determine whether the prediction model would be influenced by one ormore feature types represented by the training data. Some aspects (e.g.,features) of the training data may have a greater influence on theprediction model's results as compared to other aspects. Depending onthe results of the relevancy model, the training data may be updated tomodify (e.g., suppress or boost) the amount of the influence thesefeature types have on the prediction model's results. In this way, theprediction model's results may be improved because the prediction modelwill be trained with optimized training data. For example, the trainingdata used to train the prediction model may reduce or eliminate animpact of one or more particular feature types on the results of theprediction model. In this way, because of the reduction or eliminationof such impact, the foregoing may improve the accuracy of the predictionmodel.

In some embodiments, system 100 may obtain one or more datasets forgenerating training data to be used to train the prediction model. Insome embodiments, the datasets may be randomly selected from one or moredata corpora. The datasets may include a plurality of feature types, andeach dataset may include some or all of the plurality of feature types.In some embodiments, data including a set of different features may beprovided as an input into a prediction model to obtain a result. Thefeatures may be of a same “type” or may be of different types. As anexample, data including features, such as noise ratios, lengths ofsound, relative power, etc., may serve as an input to a prediction modelrelated to recognizing phonemes for speech recognition processes. Asanother example, data including features such as edges, objects, pixelinformation, may serve as an input to a prediction model related tocomputer vision analysis. As still yet another example, data includingfeatures, such as income, credit score, and biographical information mayserve as an input to a prediction model related to financialapplications. Each of the features (e.g., noise rations, lengths ofsound, relative power, edges, objects, income, credit score,biographical information, or other features) may be different types offeatures. The feature type may relate to the genre of the predictionmodel (e.g., speech recognition models, computer vision models, etc.) orthe different individual fields encompassed by a feature (e.g., lengthof sounds in units of time, income in units of dollars, etc.). Asdescribed herein, a feature type corresponds to a type of feature, i.e.,what the feature represents. For example, the feature type of salaryinformation corresponds to the feature salary, which may be used as aninput to a financially-related prediction model.

A dataset may include data associated with one or more different featuretypes. For example, a dataset may include financially relevant data,such as credit card applications, loan applications, or otherapplications. In one use case, the dataset may include a number ofcredit card application forms, where each such form is associated with aparticular individual or entity. The information included within thecredit card application form (e.g., name, address, age, gender, salary,etc.) may correspond to data for the individual or entity. Thecollection of the credit card application forms may encompass some orall of a dataset, and the information may encompass some or all of thedata including various feature types.

In some embodiments, the feature types represented by a dataset may behomogenous to a particular field such that the dataset may be used totrain a particular type of prediction model. As an example, a datasetmay include data associated with feature types such as automobileprices, automobile fuel economies, and automobile styles. This datasetmay be used to train a prediction model for pricing and/or selecting anautomobile. As another example, a dataset may include data associatedwith feature types such as credit scores, salary information, andbiographical information. This dataset may be used to train a predictionmodel for approving/disapproving loans. In some embodiments, the featuretypes included within a dataset may be related to different fields. Asan example, a dataset may include data associated with feature typessuch as automobile prices and credit scores.

Different feature types may influence an output of a given predictionmodel in different ways. An amount of influence a feature type has onthe prediction model's output may vary. Depending on the amount ofinfluence, a determination may be made prior to using the training datato train the prediction model as to whether the training dataunknowingly causes the prediction model's results to favor or not favora particular feature type or types. For instance, a determination shouldbe made prior to training the prediction model as to whether thetraining data includes feature types having a threshold amount ofinfluence on the prediction model's results as compared to other featuretypes. If so, a determination may also be made as to whether thosefeature types should influence the prediction model to their particularextent.

In some embodiments, prior to providing training data to the predictionmodel for training the prediction model, the training data may first beprovided to a relevancy model. The relevancy model may determine howrelevant one or more feature types are to the output of the predictionmodel. Alternatively, the training data may be provided to anirrelevancy model to determine how irrelevant one or more feature typesare to the output of the prediction model. In some embodiments, afeature type determined to have a threshold amount of influence (ormore) on the prediction model's results may indicate that the trainingdata should be updated. In some embodiments, updating the training datamay improve the accuracy and reliability of the prediction model becausethe amount of influence imparted by the feature type may be reduced. Forexample, the training data may be used to eliminate or reduce an impactof a feature type on the results of the prediction model. Therefore, ifthe relevancy model indicates that the datasets used to generate thetraining data include a feature type that has an amount of influenceequal to or greater than the threshold amount of influence, then thetraining data may be updated. In this way, because of the reduction orelimination of such impact, the foregoing may improve the predictionmodel.

In some embodiments, the training data may be generated by aggregatingthe datasets together. After aggregation, the datasets may be formatted,normalized, or processed in other manners so that the training data maybe provided to the relevancy model as an input. Continuing theaforementioned example related to credit card applications, the datasetsmay be formatted such that extraneous characters are removed, a sameinput language is used, or the like. In some embodiments, the datasetsmay be formatted by including logical zeros or null values forinformation not provided. For example, a credit card application havingthe address field left blank may be automatically filled in with alogical zero (e.g., “0”) to allow the application to be aggregated withother applications when compiling the training data.

In some embodiments, updating the training data may include obtainingone or more additional datasets. Some embodiments may include randomlyselecting the additional datasets from one or more data corpora. Eachadditional dataset may be selected from a same data corpus as the datacorpora with which the initial datasets were selected from. The randomlyselected additional datasets and at least some of the previouslyobtained datasets may be used to generate the updated training data. Theupdated training data may be provided to the relevancy model todetermine whether the relevance of the feature type has the thresholdamount of influence on the prediction model's results. If not, theupdated training data may be provided to the prediction model such thatthe prediction model may be trained on the updated training data.However, if the relevance of the feature type has the threshold amountof influence on the prediction model's results, the process of obtainingmore datasets may be repeated to generate further updated training data.

In some embodiments, a set of feature types may be identified prior toproviding the training data to the relevancy model. The set of featuretypes may include one or more feature types that should be preventedfrom having the threshold amount of influence on the prediction model,e.g., the prediction model's results. Preventing a feature type fromhaving the threshold amount of influence on the prediction model mayinclude ensuring that the amount of influence of each feature type ofthe set of feature types is less than the threshold amount of influence.In some embodiments, after the feature type has been prevented fromhaving the threshold amount of influence on the prediction model'sresults (e.g., determined to not have or have less than the thresholdamount of influence), based on the updated training data, anotherdetermination may be made. This additional determination may includeanalyzing whether one or more other feature types from the set offeature types has/have the threshold amount of influence on theprediction model. If so, then the process of obtaining datasets andgenerating updated training data may also be repeated until all featuretypes included within the set of feature types have less than thethreshold amount of influence on the prediction model.

In some embodiments, the relevancy model may include one or morestatistical relevancy models. As an example, the relevancy model mayinclude principal component analysis (PCA) model for identifying aprincipal component of a dataset. The principle component may be thefeature type that has the largest amount of variance for the trainingdata. For example, the principle component may be the feature typehaving a greatest relevance on the prediction model's results withrespect to other feature types. PCA may be thought of as fitting ann-dimensional ellipsoid to the training data. The axes of then-dimensional ellipsoid represent the principle components of thetraining data.

To determine whether the relevance of a feature type included within thetraining data will have an amount of influence on a prediction modelequal to or greater than a threshold amount of influence, a relevance ofone or more feature types may be determined via a relevancy model. Forexample, the principle component or components of the training data maybe determined via the PCA model. In some embodiments, the relevance of afeature type may be determined. If the relevance of a particular featuretype satisfies a condition, this may indicate that the training datashould be updated. In some embodiments, the condition may be whether afeature type's amount of influence is equal to or greater than athreshold amount of influence. For example, the relevancy model maycompute a relevancy score for the feature type, and the relevancythreshold score may be set a particular value (e.g., >0.01). In thisexample, if the relevancy score exceeds the relevancy threshold score,the feature type satisfies the condition. In some embodiments, thecondition may be whether the feature type's amount of influence is lessthan a threshold amount of influence. For example, if the relevancyscore is less than the relevancy threshold score, then feature type maysatisfy the condition.

Although a PCA model is described above, different relevancy models mayalternatively or additionally be used to reduce the amount of influenceof a particular feature type. For example, models includingcorrespondence analysis (CA), factor analysis, K-means clustering, andnon-negative matrix factorization may be used to identify relevantdimensions of a dataset, and furthermore for reducing the amount ofinfluence of those dimensions. Still further, models includingindependent component analysis (ICA) and network component analysis maybe used in lieu of or in addition to a PCA model for determiningrelevancy.

In some embodiments, the prediction model may include one or more neuralnetworks or other machine learning models. As an example, neuralnetworks may be based on a large collection of neural units (orartificial neurons). Neural networks may loosely mimic the manner inwhich a biological brain works (e.g., via large clusters of biologicalneurons connected by axons). Each neural unit of a neural network may beconnected with many other neural units of the neural network. Suchconnections can be enforcing or inhibitory in their effect on theactivation state of connected neural units. In some embodiments, eachindividual neural unit may have a summation function which combines thevalues of all its inputs together. In some embodiments, each connection(or the neural unit itself) may have a threshold function such that thesignal must surpass the threshold before it propagates to other neuralunits. These neural network systems may be self-learning and trained,rather than explicitly programmed, and can perform significantly betterin certain areas of problem solving, as compared to traditional computerprograms. In some embodiments, neural networks may include multiplelayers (e.g., where a signal path traverses from front layers to backlayers). In some embodiments, back propagation techniques may beutilized by the neural networks, where forward stimulation is used toreset weights on the “front” neural units. In some embodiments,stimulation and inhibition for neural networks may be more free-flowing,with connections interacting in a more chaotic and complex fashion.

As an example, with respect to FIG. 2, machine learning model 202 maytake inputs 202 and provide outputs 206. In some embodiments, outputs206 may be fed back to machine learning model 202 as input to trainmachine learning model 202 (e.g., alone or in conjunction with userindications of the accuracy of outputs 206, labels associated with theinputs, or with other reference feedback information). In someembodiments, machine learning model 202 may update its configurations(e.g., weights, biases, or other parameters) based on its assessment ofits prediction (e.g., outputs 206) and reference feedback information(e.g., user indication of accuracy, reference labels, or otherinformation). In some embodiments, where machine learning model 202 is aneural network, connection weights may be adjusted to reconciledifferences between the neural network's prediction and the referencefeedback. Some embodiments include one or more neurons (or nodes) of theneural network requiring that their respective errors are sent backwardthrough the neural network to them to facilitate the update process(e.g., backpropagation of error). Updates to the connection weights may,for example, be reflective of the magnitude of error propagated backwardafter a forward pass has been completed. In this way, for example, themachine learning model 202 may be trained to generate betterpredictions.

Subsystems 112-116

In some embodiments, training data subsystem 112 may generate trainingdata based, at least in part, on one or more datasets obtained from datacorpus database(s) 134. Data corpus database(s) 134 may include one ormore data corpora configured to store a plurality of datasets. Thedatasets may be used to create training data for training a predictionmodel. As an example, with respect to FIG. 3, data corpus databases 134may include datasets 1-M. In some embodiments, the number of datasetsstored within data corpus databases 134 may be 100 or more datasets,1,000 or more datasets, 1,000,000 or more datasets, etc.

Each dataset may be associated with one or more feature types. Forexample, dataset 1 may include data associated with N different featuretypes, dataset 2 may include data associated with L different featuretypes, and dataset M may include data associated with P differentfeature types. In some embodiments, N, L, and P are integer numbersgreater than one. In some embodiments, N, L, and P may be the same ordifferent. Furthermore, the feature types included in each of theM-datasets may overlap. For instance, one feature type representedwithin dataset 1 may also be represented within dataset 2. In someembodiments, the number of feature types (e.g., N, L, P) in theM-datasets may be 2 or more feature types, 10 or more feature types, 100or more feature types, 1,000 or more feature types, etc.

Training data subsystem 112 may obtain one or more datasets from datacorpus databases 134. In some embodiments, training data subsystem 112may be configured to randomly select the datasets from data corpusdatabases 134 in response to an instruction received from client device104. For example, client device 104 may receive a request to train, orgenerate training data for training, a prediction model. In response tothe request, client device 104 may generate and send an instructionacross network(s) 150 to computer system 102. Upon receipt of theinstruction, training data subsystem 112 may access data corpusdatabases 134 and randomly select the datasets.

Some embodiments may include randomly selecting the datasets bydetermining a random seed value via a random number generator. A datasetmay be selected based on the random seed value. For example, trainingdata subsystem 112 and/or data corpus databases 134 may include an indexof the stored datasets, e.g., datasets 1-M. The random seed value may beinput into a hash function, obtaining a hash value used to identify adataset to be selected. For instance, each dataset may be associatedwith a hash value or a range of hash values, and a dataset may beselected based on the obtained hash value for a given random seed value.After a dataset is selected, the dataset may be provided to trainingdata subsystem 112 via network(s) 150. In some embodiments, the numberof datasets selected may include 10 or more datasets, 100 or moredatasets, or 1,000 or more datasets. In some embodiments, the datasetsmay be retrieved from data corpus databases 134 serially and/or inparallel.

In some embodiments, training data subsystem 112 may be configured togenerate training data based on the randomly selected datasets obtainedfrom data corpus databases 134. The training data may include some orall of the obtained datasets. For example, each dataset obtained fromdata corpus databases 134 may be used to generate the training data.Alternatively, only some of the datasets obtained from data corpusdatabases 134 may be used to generate the training data. In someembodiments, the datasets used to generate the training data may berandomly selected from the obtained datasets.

In some embodiments, the datasets used to generate the training data mayinclude a plurality of feature types. For example, the datasets mayinclude 2 or more feature types, 10 or more feature types, 100 or morefeature types, 1,000 or more feature types, or 100,000 or more featuretypes. In some embodiments, different datasets may have a differentnumber of feature types, and some datasets may include similar featuretypes.

In some embodiments, the obtained datasets may include feature typesthat, when included within the training data used to train a predictionmodel, influence the prediction model to favor a particular result. Theamount of influence that a feature type has on the prediction model'sresults may vary, and some feature types may have more influence thanothers. In some embodiments, a determination may be made as to an amountof influence that a particular feature type has on a given predictionmodel. To perform this determination, the training data may be providedto a relevancy model prior to being used to train the prediction model.If the amount of influence, as determined using the relevancy model, ofa feature type is equal to or greater than a threshold amount ofinfluence, then the training data may be updated. The updated trainingdata may also be provided to the relevancy model to ensure that thefeature type has less than the threshold amount of influence on theprediction model. In some embodiments, the threshold amount of influencemay be predetermined. For example, a numeric value may be set as thethreshold amount of influence, e.g., 0.1. In this way, if a certainfeature type has more or less influence than desired, the training datamay not be used to train the prediction model, and updated training datamay be generated.

Depending on the prediction model, an amount of influence may becomputed for a particular set of feature types. In some embodiments, therequest received from client device 104 may indicate which predictionmodel is to be trained. Prediction models to be trained or that havebeen trained may be stored in model database(s) 138. Each predictionmodel may be stored in association with an identifier that may allowcomputer system 102 and/or client device(s) 104 to access thatprediction model. In some embodiments, the request may include theidentifier indicating the particular prediction model(s) to beretrieved. In some embodiments, model subsystem 116 may determine theprediction model to be accessed from model databases 138 based on therequest. The feature types that may be input to a prediction model mayalso be stored in model databases 138. For instance, each predictionmodel may indicate the feature types that serve as inputs.

In some embodiments, each prediction model may indicate a set of featuretypes that should not serve as an input and/or should have less than thethreshold amount of influence on the prediction model. For instance, theset of feature may be those feature types that should be prevented fromhaving the threshold amount of influence on the prediction model. If afeature type from the set of feature types is determined to have anamount of influence on the prediction model that is equal to or greaterthan a threshold amount of influence, this may indicate that thetraining data should not be used to train the prediction model. In someembodiments, if the feature type is determined to have an amount ofinfluence on the prediction model that is equal to or greater than thethreshold amount of influence, the training data may be updated. Theamount of influence the feature type has on the prediction model may bedetermined again, this time based on the updated training data. In someembodiments, the feature type(s) that should have less than thethreshold amount of influence may be pre-selected prior to beingprovided to the relevancy model. For instance, the relevancy model maybe used to determine a relevance of a first feature type based on theprediction model to be served with the input training data. In someembodiments, the feature type may be randomly selected. Furthermore, insome embodiments, multiple feature types may be checked via therelevancy model in parallel.

In some embodiments, the training data may be provided to relevancysubsystem 114. Furthermore, an indication of a particular feature typeor feature types with which a relevance is to be determined may beprovided to relevancy subsystem 114. Some embodiments may determine arelevance for each feature type the training data includes. Relevancysubsystem 114 may obtain a relevancy model from model database(s) 138.Relevancy subsystem 114 may input the training data to the relevancymodel and may determine a relevance of one (or more) of the featuretypes. As an example, the relevancy model may be the PCA model, and thetraining data may be analyzed to determine the principle component.

In some embodiments, relevancy subsystem 114 may determine whether therelevance satisfies a condition, such as if an amount of influence afeature type would have on a prediction model's results if trained onthe training data is equal to or greater than a threshold amount ofinfluence. As an example, relevancy subsystem 114 may compute, via therelevancy model, a relevancy score associated with a feature type basedon the training data. Relevancy subsystem 114 may compare the relevancyscore to a relevancy threshold score. If the relevancy score is equal toor greater than the relevancy threshold score, this may indicate thatthe prediction model should not be trained using the training data. Forinstance, if the training data was used to train the prediction model,the feature type may have an undesired amount of influence on theprediction model's results (e.g., favoring a particular outcome). Inresponse to determining that the relevancy of the feature type satisfiesthe condition, relevancy subsystem 114 may generate an instruction orother indication to cause training data subsystem 112 to obtain one ormore additional datasets for updating the training data.

After receiving the indication that the additional datasets are to beobtained, training data subsystem 112 may access data corpus database(s)134. Similarly to the previously described process for obtaining thedatasets used to generate the training data, the additional datasets mayalso be randomly selected from data corpus database(s) 134. Trainingdata subsystem 112 may be configured to generate updated training databased on the additional datasets and the previously obtained datasets.In some embodiments, the updated training data may include at least someof the additional datasets and at least some of the previously obtaineddatasets.

In some embodiments, the additional datasets may include some or all ofthe plurality of feature types that the previously obtained datasetsincluded. For example, if the previously obtained datasets included dataassociated with feature type A and feature type B, the additionaldatasets may also include data associated with feature type A andfeature type B. In some embodiments, the additional datasets may includedata associated with additional feature types. For example, theadditional datasets may also include data associated with feature typeC. Therefore, the updated training data may include data associated withfeature types A and B, as well as feature type C.

After the updated training data is generated, relevancy subsystem 114may input the updated training data to the relevancy model. In someembodiments, relevancy subsystem 114 may be configured to determinewhether an updated relevance of the feature type satisfies an additionalcondition. The additional condition may relate to whether the featuretype has the threshold amount of influence (or more) on the predictionmodel if the prediction model were trained using the updated trainingdata. In some embodiments, the feature type analyzed against theadditional condition is the same feature type previously analyzed. Forexample, the feature type having a relevance previously determined tosatisfy the condition, e.g., that the feature type has an amount ofinfluence on the prediction model's results equal to or greater than thethreshold amount of influence, may be checked to determine whether anupdated relevance of the feature type satisfies the additionalcondition, e.g., that the feature type has an updated amount ofinfluence on the prediction model's results less than the thresholdamount of influence.

In some embodiments, the training data generated based on the previouslyobtained datasets including a plurality of feature types may be input tothe relevancy model, and a determination may be made whether a featuretype of the plurality of feature types satisfies a first condition. Thefirst condition may indicate whether the feature type will have thethreshold amount of influence on the prediction model if trained on thetraining data. If the feature type is determined to satisfy the firstcondition, the updated training data may be generated based on theadditional datasets including the plurality of feature types. Afterbeing generated, the updated training data may be provided as an inputto the relevancy model, and a determination may be made whether thefeature type satisfies a second condition. The second condition mayindicate whether the feature type will not have the threshold amount ofinfluence (e.g., has less than the threshold amount of influence) on theprediction model if trained on the updated training data. If the featuretype satisfies the second condition, the updated training data may beprovided to the prediction model so that the prediction model may betrained on the updated training data. If the feature type does notsatisfy the second condition, then the training data may again beupdated by repeating the foregoing process.

As an example, relevancy subsystem 114 may compute, via the relevancymodel, a relevancy score associated with a feature type based on thetraining data. Relevancy subsystem 114 may then compare the relevancyscore to a relevancy threshold score. If the relevancy score is equal toor greater than the relevancy threshold score, this may indicate thatthe prediction model should not be trained using the training data. Forinstance, if the training data was used to train the prediction model,the feature type could have an undesired amount of influence on theprediction model's results. In response to determining that therelevancy of the feature type satisfies the condition, relevancysubsystem 114 may generate an instruction or other indication to causetraining data subsystem 112 to obtain one or more additional datasetsfor updating the training data.

In some embodiments, model subsystem 116 may be configured to receivetraining data from relevancy subsystem 114. The training data may bereceived by model subsystem 116 subsequent to relevancy subsystem 114determining that no feature type of a predetermined set of feature typeshas the threshold amount of influence (or more) on the predictionmodel's results if the prediction model were to be trained using thereceived training data. For example, as previously mentioned, eachprediction model may include a set of feature types that should not havemore a threshold amount of influence on the prediction model's results.Relevancy subsystem 114 may determine whether particular training datasatisfies such criteria, and upon doing so, may provide the trainingdata to model subsystem 116.

In some embodiments, model subsystem 116 may provide the training data,e.g., the training data that does not include any feature type from theset of feature types having the threshold amount of influence, totraining data database(s) 136. The training data may be stored bytraining data database(s) 136 to be used to train the prediction model.In some embodiments, model subsystem 116 may alternatively oradditionally provide the training data to the prediction model fortraining the prediction model. For example, model subsystem 116 mayobtain the prediction model from model database(s) 138. The receivedtraining data may input to the prediction model to train the predictionmodel, or the training data may be retrieved from training datadatabase(s) 136 and then input to the prediction model in order to trainthe prediction model. In some embodiments, the training data stored intraining data database(s) 136 may include a timestamp indicating acreation time for that training data. Furthermore, the datasets includedin the training data and the feature types analyzed by the relevancymodel for the training data may also be stored in association with thetraining data for subsequent/additional prediction model training.

FIG. 4A shows a relevancy model configured to determine a relevance of afeature type based on input training data, in accordance with one ormore embodiments. In some embodiments, training data 402 may begenerated based on one or more datasets randomly obtained from datacorpus database(s) 134. The randomly obtained datasets may include aplurality of features. Training data 402 may be provided to relevancymodel 404 to determine a relevance of at least one of the plurality offeatures. In some embodiments, the feature(s) whose relevance is to bedetermined may be included in a set of feature types that should haveless than a threshold amount of influence on results obtained by amachine learning model 410 if machine learning model 410 were trainedusing training data 402. In some embodiments, relevancy model 404computes a relevancy score for the feature type, e.g., one of thefeature types from the set of feature types. If the relevancy score forthe feature type is less than a relevancy threshold score, then thetraining data 402 may be provided to machine learning model 410 fortraining. The relevancy score for the feature type being less than therelevancy threshold score may indicate that the feature type has lessthan a threshold amount of influence of the results of machine learningmodel 410. Therefore, so long as no other feature types included in theset of feature types represented by training data 402 have equal to orgreater than the threshold amount of influence, training data 402 may beused to train machine learning model 410.

In some embodiments, the relevancy threshold score may be a numericalvalue. For example, the relevancy threshold score may be 0.1, 0.01,0.001, etc. If the relevancy score for a feature type is determined tobe 0.09 via relevancy model 404, and the relevancy threshold score isset as 0.1, then the feature type has less than the threshold amount ofinfluence on machine learning model 410. In this scenario, the trainingdata (e.g., training data 402) may be provided to machine learning model410 for training machine learning model 410.

On the other hand, as seen with reference to FIG. 4B, if the relevancyscore for a feature type is determined to be 0.12 via relevancy model404, and the relevancy threshold score is set as 0.1, then the featuretype may have more than a threshold amount of influence on machinelearning model 410. In this scenario, one or more additional datasetsmay be randomly selected from data corpus database(s) 134. Theadditional datasets and the previously obtained datasets (used togenerate training data 402) may be used to generate updated trainingdata 406. In some embodiments, updated training data 406 may be providedas an input to relevancy model 404, which may compute an updatedrelevancy score for the feature type. If the relevancy score still isequal to or greater than the relevancy threshold score, then the processof obtaining datasets and updating the training data may be repeated.However, if the relevancy score is now less than the relevancy thresholdscore, updated training data 406 may be provided to machine learningmodel 410.

Example Flowcharts

FIGS. 5 and 6 are example flowcharts of processing operations of methodsthat enable the various features and functionality of the system asdescribed in detail above. The processing operations of each methodpresented below are intended to be illustrative and non-limiting. Insome embodiments, for example, the methods may be accomplished with oneor more additional operations not described, and/or without one or moreof the operations discussed. Additionally, the order in which theprocessing operations of the methods are illustrated (and describedbelow) is not intended to be limiting.

In some embodiments, the methods may be implemented in one or moreprocessing devices (e.g., a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information). The processingdevices may include one or more devices executing some or all of theoperations of the methods in response to instructions storedelectronically on an electronic storage medium. The processing devicesmay include one or more devices configured through hardware, firmware,and/or software to be specifically designed for execution of one or moreof the operations of the methods.

FIG. 5 shows a flowchart of a method 500 of facilitating predictionmodel training, in accordance with one or more embodiments. In anoperation 502, training data including first datasets may be obtained.The first datasets may include a plurality of feature types. As anexample, one or more first datasets may be randomly obtained from one ormore data corpora. The randomly obtained first datasets may include aplurality of features, and may be used to generate training data fortraining a prediction model. In some embodiments, operation 502 may beperformed by a subsystem that is the same or similar to training datasubsystem 112.

In an operation 504, a determination may be made, via a relevancy modeland based on the generated training data, as to whether a feature typesatisfies a first condition. The first condition may relate to thefeature type having a threshold amount of influence on a machinelearning model. The feature type may be one of a plurality of featuretypes that the datasets include. As an example, the training data may beprovided to a relevancy model, and the relevancy model may determine arelevancy score for a feature type included by the training data. If therelevancy score is equal to or greater than a relevancy threshold score,the feature type may be considered to have a threshold amount ofinfluence on a prediction model to be trained on the training data. Ifthe relevancy score is less than the relevancy threshold score, thefeature type may be considered to not have (e.g., considered to haveless than) the threshold amount of influence on the prediction model tobe trained on the training data. In some embodiments, operation 504 maybe performed by a subsystem that is the same or similar to relevancysubsystem 114.

In an operation 506, responsive to determining that the feature typesatisfies the first condition, one or more second datasets may beobtained to update the training data. The updated training data mayinclude the second datasets, where the second datasets include theplurality of feature types. As an example, if the relevancy score forthe feature type is equal to or greater than the relevancy thresholdscore, one or more additional datasets may randomly be obtained from thedata corpora. The additional datasets and the previously obtaineddatasets may be used to generate updated training data, which mayinclude the plurality of feature types. In some embodiments, the updatedtraining data may include one or more additional feature types. In someembodiments, operation 506 may be performed by a subsystem that is thesame or similar to training data subsystem 112.

In an operation 508, a determination may be made, via the relevancymodel, and based on the updated training data, as to whether the featuretype satisfies a second condition. The second condition may relate tothe feature type not having the threshold amount of influence on themachine learning model. As an example, the updated training data may beprovided as an input to the relevancy model. The relevancy model maydetermine an updated relevancy score for the feature type included bythe training data. If the updated relevancy score is equal to or greaterthan the relevancy threshold score, the feature type may be consideredto have the threshold amount of influence on the prediction model to betrained on the updated training data. If the updated relevancy score isless than the relevancy threshold score, the feature type may beconsidered to not have (e.g., considered to have less than) thethreshold amount of influence on the prediction model to be trained onthe updated training data. In some embodiments, operation 508 may beperformed by a subsystem that is the same or similar to relevancysubsystem 114.

In an operation 510, responsive to determining that the feature typesatisfies the second condition, the updated training data may beprovided to the machine learning model to train the machine learningmodel. As an example, after determining that the updated relevancy scorefor the feature type is less than the relevancy threshold score, theupdated training data may be provided to the prediction model. Theupdated training data may be used to train the prediction model. In someembodiments, operation 510 may be performed by a subsystem that is thesame or similar to model subsystem 116.

FIG. 6 shows a flowchart of a method 600 for facilitating predictionmodel training, in accordance with one or more embodiments. In anoperation 602, first datasets may be randomly obtained to generatetraining data such that the training data includes the randomly obtainedfirst datasets. The first datasets may include a plurality of featuretypes. As an example, first datasets may be randomly obtained from adata corpus. Training data for training a prediction model may begenerated such that the training data includes the randomly obtainedfirst datasets. The randomly obtained first datasets may include aplurality of feature types. In some embodiments, operation 602 may beperformed by a subsystem that is the same or similar to training datasubsystem 112.

In an operation 604, the training data may be provided to a relevancymodel to determine a relevance of each feature type of the plurality offeature types with respect to an output of a prediction model if theprediction model is trained on the training data. As an example, thetraining data may be provided as an input to a relevancy model, and therelevancy model may determine a relevancy score for each feature type ofthe plurality of feature types. The relevancy score may indicate howrelevant each feature type is to an output of a machine learning modelif that machine learning model were provided with and trained on thetraining data. In some embodiments, operation 604 may be performed by asubsystem that is the same or similar to relevancy subsystem 114.

In an operation 606, responsive to determining that the relevance of afirst feature type of the plurality of feature types satisfies a firstcondition, additional datasets may be randomly obtained to update thetraining data. The updated training data may include the additionaldatasets and at least some of the first datasets. The first conditionmay indicate that the first feature type will have a threshold amount ofinfluence on the prediction model if the prediction model is trained onthe training data. The additional datasets may include the plurality offeature types. As an example, if the relevancy score of a first featuretype, as determine via the relevancy model, is equal to or greater thana relevancy threshold score, the first feature type may be considered tohave a threshold amount of influence on a machine learning model to betrained on the training data. In this scenario, additional datasetsincluding the plurality of feature types may be randomly obtained fromthe data corpus such that updated training data may be generated. If therelevancy score is less than the relevancy threshold score, the firstfeature type may be considered to not have, or have less than, thethreshold amount of influence on the machine learning model to betrained on the training data. In some embodiments, operation 606 may beperformed by a subsystem that is the same or similar to relevancysubsystem 114. In some embodiments, operation 606 may be performed by asubsystem that is the same or similar to training data subsystem 112. Insome embodiments, operation 606 may be performed by subsystems the sameor similar to training data subsystem 112 and relevancy subsystem 114.

In an operation 608, the updated training data may be provided to therelevancy model to determine an updated relevance of each feature typeof the plurality of feature types with respect to an output of theprediction model if the prediction model is trained on the updatedtraining data. As an example, the updated training data may be providedto the relevancy model. The relevancy model may determine an updatedrelevancy score for each feature type of the plurality of feature typesincluded by the updated training data with respect to the machinelearning model. In some embodiments, operation 608 may be performed by asubsystem that is the same or similar to relevancy subsystem 114.

In an operation 610, responsive to determining that the updatedrelevance of the first feature type satisfies a second condition, theupdated training data may be provided to the prediction model to trainthe prediction model. The second condition may indicate that the firstfeature type will not have the threshold amount of influence on themachine learning model if the machine learning model is trained on theupdated training data. As an example, if the updated relevancy score isequal to or greater than the relevancy threshold score, the feature typemay be considered to have the threshold amount of influence on theprediction model to be trained on the updated training data. If theupdated relevancy score is less than the relevancy threshold score, thefeature type may be considered to not have, or have less than, thethreshold amount of influence on the prediction model to be trained onthe updated training data. If the updated relevancy score is less thanthe relevancy score threshold, the updated training data may be providedto the machine learning model for training the machine learning model.In some embodiments, operation 610 may be performed by a subsystem thatis the same or similar to model subsystem 116. In some embodiments,operation 610 may be performed by a subsystem that is the same orsimilar to relevancy subsystem 114. In some embodiments, operation 610may be performed by subsystems the same or similar to relevancysubsystem 114 and model subsystem 116.

In some embodiments, the various computers and subsystems illustrated inFIG. 1 may include one or more computing devices that are programmed toperform the functions described herein. The computing devices mayinclude one or more electronic storages (e.g., prediction database(s)132, which may include data corpus database(s) 134, training datadatabase(s) 136, model database(s) 138, etc., or other electronicstorages), one or more physical processors programmed with one or morecomputer program instructions, and/or other components. The computingdevices may include communication lines or ports to enable the exchangeof information with one or more networks (e.g., network(s) 150) or othercomputing platforms via wired or wireless techniques (e.g., Ethernet,fiber optics, coaxial cable, WiFi, Bluetooth, near field communication,or other technologies). The computing devices may include a plurality ofhardware, software, and/or firmware components operating together. Forexample, the computing devices may be implemented by a cloud ofcomputing platforms operating together as the computing devices.

The electronic storages may include non-transitory storage media thatelectronically stores information. The storage media of the electronicstorages may include one or both of (i) system storage that is providedintegrally (e.g., substantially non-removable) with servers or clientdevices or (ii) removable storage that is removably connectable to theservers or client devices via, for example, a port (e.g., a USB port, afirewire port, etc.) or a drive (e.g., a disk drive, etc.). Theelectronic storages may include one or more of optically readablestorage media (e.g., optical disks, etc.), magnetically readable storagemedia (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.),electrical charge-based storage media (e.g., EEPROM, RAM, etc.),solid-state storage media (e.g., flash drive, etc.), and/or otherelectronically readable storage media. The electronic storages mayinclude one or more virtual storage resources (e.g., cloud storage, avirtual private network, and/or other virtual storage resources). Theelectronic storage may store software algorithms, information determinedby the processors, information obtained from servers, informationobtained from client devices, or other information that enables thefunctionality as described herein.

The processors may be programmed to provide information processingcapabilities in the computing devices. As such, the processors mayinclude one or more of a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information. In someembodiments, the processors may include a plurality of processing units.These processing units may be physically located within the same device,or the processors may represent processing functionality of a pluralityof devices operating in coordination. The processors may be programmedto execute computer program instructions to perform functions describedherein of subsystems 112-116 or other subsystems. The processors may beprogrammed to execute computer program instructions by software;hardware; firmware; some combination of software, hardware, or firmware;and/or other mechanisms for configuring processing capabilities on theprocessors.

It should be appreciated that the description of the functionalityprovided by the different subsystems 112-116 described herein is forillustrative purposes, and is not intended to be limiting, as any ofsubsystems 112-116 may provide more or less functionality than isdescribed. For example, one or more of subsystems 112-116 may beeliminated, and some or all of its functionality may be provided byother ones of subsystems 112-116. As another example, additionalsubsystems may be programmed to perform some or all of the functionalityattributed herein to one of subsystems 112-116.

Although example embodiments have been described in detail for thepurpose of illustration, it is to be understood that such detail issolely for that purpose and that embodiments are not limited to thedisclosed embodiments, but, on the contrary, are intended to covermodifications and equivalent arrangements that are within the scope ofthe appended claims. For example, it is to be understood thatembodiments contemplate that, to the extent possible, one or morefeatures of any embodiment can be combined with one or more features ofany other embodiment.

As used throughout this application, the word “may” is used in apermissive sense (i.e., meaning having the potential to), rather thanthe mandatory sense (i.e., meaning must). The words “comprise,”“comprising,” “comprises,” “include”, “including”, and “includes” andthe like mean including, but not limited to. As used throughout thisapplication, the singular forms “a,” “an,” and “the” include pluralreferents unless the context clearly indicates otherwise, andnotwithstanding the use of other terms and phrases for one or moreelements, such as “one or more.” The term “or” is non-exclusive (i.e.,encompassing both “and” and “or”), unless the context clearly indicatesotherwise. Further, unless otherwise indicated, statements that onevalue or action is “based on” another condition or value encompass bothinstances in which the condition or value is the sole factor andinstances in which the condition or value is one factor among aplurality of factors. Unless the context clearly indicates otherwise,statements that “each” instance of some collection have some propertyshould not be read to exclude cases where some otherwise identical orsimilar members of a larger collection do not have the property, i.e.,each does not necessarily mean each and every.

Additional example embodiments are provided with reference to thefollowing enumerated embodiments:

1. A method comprising: obtaining training data comprising one or morefirst datasets; determining (e.g., via a first model), based on thetraining data, whether a feature type satisfies a first condition;responsive to determining that the feature type satisfies the firstcondition, obtaining one or more second datasets to update the trainingdata such that the updated training data comprises the one or moresecond datasets; determining (e.g., via the first model), based on theupdated training data, whether the feature type satisfies a secondcondition; and responsive to determining that the feature type satisfiesthe second condition, providing the updated training data to a secondmodel.2. The method of embodiment 1, wherein the first condition is related tothe feature type having a threshold amount of influence on the secondmodel, the second condition is related to the feature type not havingthe threshold amount of influence on the second model.3. The method of any of embodiments 1-2, wherein the first modelcomprises a relevancy model, and the second model comprises a predictionmodel.4. The method of embodiment 3, wherein the prediction model comprises amachine learning model.5. The method of embodiment 4, wherein the machine learning modelcomprises a neural network.6. The method of any of embodiments 1-5, wherein the updated trainingdata comprises the one or more second datasets and one or more of theone or more first datasets.7. The method of embodiment 6, wherein the updated training data doesnot comprise one or more other ones of the one or more first datasets.8. The method of any of embodiments 1-7, wherein: obtaining the trainingdata comprising the one or more first datasets comprises randomlyobtaining the one or more first datasets from a data corpus to generatethe training data such that the training data comprises the randomlyobtained first datasets; and obtaining the one or more second datasetscomprises randomly obtaining additional datasets from the data corpus toupdate the training data such that the updated training data comprisesthe randomly obtained additional datasets and at least some of therandomly obtained first datasets.9. The method of any of embodiments 1-8, wherein: determining, via thefirst model, based on the training data, whether the feature typesatisfies the first condition comprises: providing the training data tothe first model to determine a relevance of each feature type of aplurality of feature types with respect to an output of the second modelif the second model is training on the training data; and determining,via the first model, based on the updated training data, whether thefeature type satisfies the second condition comprises providing theupdated training data to the first model to determine an updatedrelevance of each feature type of the plurality of feature types withrespect to an output of the second model if the second model is trainedon the updated training data.10. The method of any of embodiments 1-9, wherein: the relevancecomprises a relevancy score; the updated relevance comprises an updatedrelevancy score; the relevancy score and the updated relevancy score arecomputed by the first model, wherein the first model comprises arelevancy model; the threshold amount of influence comprises a relevancythreshold score; the relevance of the first feature type beingdetermined to satisfy the first condition comprises determining that therelevancy score is equal to or greater than the relevancy thresholdscore; and the updated relevance of the first feature type beingdetermined to satisfy the second condition comprises determining thatthe updated relevancy score is less than the relevancy threshold score.11. The method of any of embodiments 1-10, further comprising:identifying a set of feature types that are to be prevented from havingthe threshold amount of influence on the second model, wherein the setof feature types comprises the feature type.12. The method of embodiment 11, wherein the method further comprises:preventing each feature type of the set of feature types from having thethreshold amount of influence on the second model.13. The method of embodiment 12, wherein preventing comprises:determining, prior to providing the training data or the updatedtraining data to the second model, that a relevance or an updatedrelevance of each feature type included within the set of feature typeshas an amount of influence on the second model that is less than thethreshold amount of influence.14. The method of any of embodiments 11-13, wherein the set of featuretypes further comprises at least one additional feature type, and theupdated training data is provided to the second model responsive todetermining, based on the updated training data provided to the firstmodel, that the updated relevance of each of the first feature type andthe at least one additional feature type satisfies the second condition.15. The method of embodiment 14, wherein the one or more first datasetscomprises a plurality of feature types, and the plurality of featuretypes comprises the at least one additional feature type.16. The method of embodiment 14, wherein the one or more first datasetscomprises a plurality of feature types, and wherein the plurality offeature types does not comprise the at least one additional featuretype.17. The method of embodiment 14, further comprising: determining, priorto providing the updated training data to the second model, that eachfeature type of the set of feature types satisfies the second condition.18. The method of any of embodiments 1-17, wherein: obtaining thetraining data comprises randomly selecting the one or more firstdatasets from one or more data corpora; and wherein obtaining the one ormore second datasets comprises randomly selecting the one or more seconddatasets from the one or more data corpora.19. The method of embodiment 18, wherein: obtaining the training datacomprises generating the training data using the randomly selected firstdatasets; and wherein obtaining the one or more randomly selected seconddatasets comprises generating the updated training data.20. The method of any of embodiments 1-19, wherein the one or more firstdatasets comprise a first number of datasets and the one or more seconddatasets comprise a second number of datasets, and the first number ofdatasets is equal to, less than, or greater than the second number ofdatasets.21. The method of any of embodiments 1-20, wherein the threshold amountof influence comprises a relevancy threshold score, determining whetherthe feature type satisfies the first condition comprises: determiningwhether a relevancy score for the feature type is equal to or greaterthan the relevancy threshold score.22. The method of embodiment 21, wherein the relevancy score being equalto or greater than the relevancy threshold score indicates that thefeature type would have the threshold amount of influence on the secondmodel if the second model were trained on the training data.23. The method of embodiment 22, wherein determining whether the featuretype satisfies the second condition comprises: determining whether anupdated relevancy score for the feature type is less than the relevancythreshold score.24. The method of embodiment 23, wherein the updated relevancy scorebeing less than the relevancy threshold score indicates that the featuretype would not have the threshold amount of influence on the secondmodel if the second model were trained on the updated training data.25. The method of any of embodiments 1-24, wherein the one or more firstdatasets comprise a plurality of feature types, the additional datasetscomprise the plurality of feature types, and the plurality of featuretypes comprise the feature type.26. The method of embodiment 25, wherein the additional datasets furthercomprise one or more additional feature types.27. The method of any of embodiments 1-26, further comprising: causingthe second model to be trained based on the updated training data.28. The method of any of embodiments 1-27, further comprising: receivinga request to generate training data to train the second model; andselecting the second model from a plurality of prediction models basedon the request.29. One or more tangible, non-transitory, machine-readable media storinginstructions that, when executed by one or more processors, effectuationoperations comprising those of any of embodiments 1-28.30. A system comprising: one or more processors; and memory storingcomputer program instructions that, when executed by the one or moreprocessors, cause the one or more processors to effectuate operationscomprising those of any of embodiments 1-28.

What is claimed is:
 1. A non-transitory computer-readable medium storingcomputer program instructions that, when executed by one or moreprocessors, effectuate operations comprising: obtaining first trainingdata comprising first datasets, wherein the first datasets comprise aplurality of feature types; determining, via a relevancy model, based onthe first training data, whether a feature type of the plurality offeature types satisfies a first condition, the first condition beingsatisfied comprising a relevancy score of the feature type being equalto or greater than a threshold relevancy score; responsive todetermining that the feature type satisfies the first condition,obtaining one or more second datasets to update the first training datato obtain second training data, wherein the second training datacomprises the one or more second datasets, wherein the one or moresecond datasets comprise the plurality of feature types; determining,via the relevancy model, based on the second training data, whether thefeature type satisfies a second condition, the second condition beingsatisfied comprising an updated relevancy score of the feature typebeing less than the threshold relevancy score; and responsive todetermining that the feature type satisfies the second condition,causing a machine learning model to be trained with the second trainingdata.
 2. The non-transitory computer-readable medium of claim 1, whereinthe second training data does not comprise one or more other ones of thefirst datasets.
 3. The non-transitory computer-readable medium of claim1, wherein the operations further comprise: identifying a set of featuretypes that are to be prevented from having a threshold amount ofinfluence on the machine learning model, wherein the set of featuretypes comprises the feature type.
 4. The non-transitorycomputer-readable medium of claim 3, wherein the operations furthercomprise: determining, prior to the second training data being used totrain the machine learning model, that each feature type of the set offeature types satisfies the second condition.
 5. The non-transitorycomputer-readable medium of claim 1, wherein: obtaining the firsttraining data comprises randomly selecting the first datasets from oneor more data corpora; and obtaining the one or more second datasetscomprises randomly selecting the one or more second datasets from theone or more data corpora.
 6. The non-transitory computer-readable mediumof claim 1, wherein: the machine learning model comprises a neuralnetwork; and the relevancy model comprises a principle componentanalysis (PCA) model.
 7. The non-transitory computer-readable medium ofclaim 1, wherein the operations further comprise: determining, via therelevancy model, based on the second training data, whether anadditional feature type of the plurality of feature types satisfies thefirst condition; responsive to determining that the additional featuretype satisfies the first condition, obtaining one or more third datasetsto update the second training data to obtain third training data; anddetermining, via the relevancy model, based on the third training data,whether the additional feature type satisfies the second condition,wherein whether the feature type satisfies the second condition isdetermined responsive to the additional feature type being determined tosatisfies the second condition.
 8. A system, comprising: memory storingcomputer program instructions; and one or more processors that, inresponse to executing the computer program instructions, effectuateoperations comprising: obtaining first training data comprising firstdatasets, wherein the first datasets comprise a plurality of featuretypes; determining, via a relevancy model, based on the first trainingdata, whether a feature type of the plurality of feature types satisfiesa first condition, the first condition being satisfied comprising arelevancy score of the feature type being equal to or greater than athreshold relevancy score; responsive to determining that the featuretype satisfies the first condition, obtaining one or more seconddatasets to update the first training data to obtain second trainingdata, wherein the second training data comprises the one or more seconddatasets, wherein the one or more second datasets comprise the pluralityof feature types; determining, via the relevancy model, based on thesecond training data, whether the feature type satisfies a secondcondition, the second condition being satisfied comprising an updatedrelevancy score of the feature type being less than the thresholdrelevancy score; and responsive to determining that the feature typesatisfies the second condition, causing a machine learning model to betrained with the second training data.
 9. The system of claim 8, whereinthe second training data does not comprise one or more other ones of thefirst datasets.
 10. The system of claim 8, wherein the operationsfurther comprise: identifying a set of feature types that are to beprevented from having a threshold amount of influence on the machinelearning model, wherein the set of feature types comprises the featuretype.
 11. The system of claim 10, wherein the operations furthercomprise: determining, prior to the second training data being used totrain the machine learning model, that each feature type of the set offeature types satisfies the second condition.
 12. The system of claim 8,wherein: obtaining the first training data comprises randomly selectingthe first datasets from one or more data corpora; and obtaining the oneor more second datasets comprises randomly selecting the one or moresecond datasets from the one or more data corpora.
 13. The system ofclaim 8, wherein: the machine learning model comprises a neural network;and the relevancy model comprises a principle component analysis (PCA)model.
 14. The system of claim 8, wherein the operations furthercomprise: determining, via the relevancy model, based on the secondtraining data, whether an additional feature type of the plurality offeature types satisfies the first condition; responsive to determiningthat the additional feature type satisfies the first condition,obtaining one or more third datasets to update the second training datato obtain third training data; and determining, via the relevancy model,based on the third training data, whether the additional feature typesatisfies the second condition, wherein whether the feature typesatisfies the second condition is determined responsive to theadditional feature type being determined to satisfies the secondcondition.
 15. A method implemented by one or more processors configuredto execute computer program instructions, the method comprising:obtaining first training data comprising first datasets, wherein thefirst datasets comprise a plurality of feature types; determining, via arelevancy model, based on the first training data, whether a featuretype of the plurality of feature types satisfies a first condition, thefirst condition being satisfied comprising a relevancy score of thefeature type being equal to or greater than a threshold relevancy score;responsive to determining that the feature type satisfies the firstcondition, obtaining one or more second datasets to update the firsttraining data to obtain second training data, wherein the secondtraining data comprises the one or more second datasets, wherein the oneor more second datasets comprise the plurality of feature types;determining, via the relevancy model, based on the second training data,whether the feature type satisfies a second condition, the secondcondition being satisfied comprising an updated relevancy score of thefeature type being less than the threshold relevancy score; andresponsive to determining that the feature type satisfies the secondcondition, causing a machine learning model to be trained with thesecond training data.
 16. The method of claim 15, further comprising:identifying a set of feature types that are to be prevented from havinga threshold amount of influence on the machine learning model, whereinthe set of feature types comprises the feature type.
 17. The method ofclaim 16, further comprising: determining, prior to the second trainingdata being used to train the machine learning model, that each featuretype of the set of feature types satisfies the second condition.
 18. Themethod of claim 15, wherein: obtaining the first training data comprisesrandomly selecting the first datasets from one or more data corpora; andobtaining the one or more second datasets comprises randomly selectingthe one or more second datasets from the one or more data corpora. 19.The method of claim 15, wherein: the second training data does notcomprise one or more other ones of the first datasets; the machinelearning model comprises a neural network; and the relevancy modelcomprises a principle component analysis (PCA) model.
 20. The method ofclaim 15, further comprising: determining, via the relevancy model,based on the second training data, whether an additional feature type ofthe plurality of feature types satisfies the first condition; responsiveto determining that the additional feature type satisfies the firstcondition, obtaining one or more third datasets to update the secondtraining data to obtain third training data; and determining, via therelevancy model, based on the third training data, whether theadditional feature type satisfies the second condition, wherein whetherthe feature type satisfies the second condition is determined responsiveto the additional feature type being determined to satisfies the secondcondition.